There is a technique that generates a multi-viewpoint image by calculating a disparity from a stereo image by a stereo matching technique, and the like and generating an intermediate image by using the disparity information.
In recent years, has been disclosed an image warping based multi-viewpoint image generating technique that generates a multi-viewpoint image by mesh grid based image warping type calculation to minimize an energy function defined by calculating disparity information only for a partial area of the image through calculation of a feature disparity/optical flow in the stereo image, defining each item of an energy function by using a restriction condition defined as image saliency and image edge components, and weight combining the respective items.
However, a phenomenon occurs, in which distortion of an entire image is crude by warping at the time of generating the multi-viewpoint image by the warping by using the aforementioned technique, and as a result, person objects are dropped and bent to increase inaccuracy of the multi-viewpoint image. Further, a position and a degree in which the distortion occurs vary for each multi-viewpoint image. This generates the distortion be further remarkable at the time of sequentially reproducing the multi-viewpoint images. In particular, in an existing method that generates an intermediate viewpoint image by using an image in a close direction between left and right images, a remarkable difference occurs between the intermediate viewpoint images positioned at the center by a difference of the left and right images. This has a disadvantage of degrading reliability of the multi-viewpoint image and concentrativeness of a viewer at the time of switching the multi-viewpoint images.
Accordingly, a similarity between contiguous viewpoint images is increased so that switching the multi-viewpoint images is natural at the time of generating the multi-viewpoint image and the direction of the distortion is consistent and a degree of the distortion needs to be evenly distributed among the multi-viewpoint images so that the distortion which occurs in the multi-viewpoint images while warping the image is natural from the viewer side.